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DETAILED ACTION 

Response to Arguments 

1 . Applicant's arguments filed on 3/12/2009 have been fully considered but they are 
not persuasive. 

On page 13, applicant argues that none of the cited references teach 
manipulating information in live video images. The examiner disagrees; "Manipulating a 
visual representation of the information in the first live video image and in the second 
live video image" is taught by Saka (see fig. 12 and pg. 4 sect. 0042). Saka teaches 
exact images of two different desktops being shown side-by-side. When a user wants 
to transfer information, he drags (i.e. manipulates) an icon (i.e. a visual representation 
of the information) from a live view of one desktop (a first live video image) to a live view 
of a second desktop (a second live video image). Both of the devices have hotspots 
(icons) on the respective views of their desktops that, when manipulated or accessed, 
cause the associated information to have some action done to it. The term "live video 
image" includes the image of the client's desktop because the desktop image is a 
current and updated (i.e. live) appearance of text and graphics on a video display 
(which is definition of "video" as defined in The American Heritage Dictionary of the 
English Language). 

Claim 1 0 contains amendments similar to those of claim 1 and the above 
argument made in regard to claim 1 also applies to claim 10. Claim 10 limitation 
requiring the hotspot to be dynamically defined is also met by Saka because the icon 
(hotspot) can be moved. 
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Applicant argues on page 15 of his remarks that claim 18, as amended, is not 
taught by Ludwig in view of Schneiderman. The examiner disagrees; Schneiderman 
teaches annotating an "image aspect" (col. 7 II. 4-9), where an image aspect is defined 
as "a scene, object, event, design, etc" (col. 7 II. 23-25). A "device", by definition, 
encompasses any thing made for a particular purpose. Therefore any object that has a 
purpose qualifies as a device. Although Schneiderman uses a person as an illustrative 
example, he makes it clear that his invention does not only apply to annotating images 
of people. Furthermore, Schneiderman's annotation is transferred to a particular image 
aspect, or person in the illustrative example. It is clear that the annotation is not 
intended to apply to the entire image, but to the image aspect which it is drawn over 
(see col. 7 II. 53-54 "Fig. 7 shows the annotation of four identified individuals"). In fig. 5, 
the annotation of Rae Earnshaw applies only to the portion of the image that is Rae 
Earnshaw. The user choosing an X,Y location in the image is a means to associate the 
annotation to Rae, since Rae's X,Y location is a way of representing him. 
Schneiderman transfers Rae's annotation to him by transferring it to an X,Y location that 
is associated with Rae. 

However, the examiner notes that applicant's arguments are addressed toward a 
moving image and introduces US Patent 6,71 1 ,590 to Lennon. Lennon teaches a 
technique to track and link annotations to objects in a moving video (abstract, fig. 4 and 
col. 10 II. 61 -col. 11 II. 5). It would have been obvious to use the technology taught by 
Lennon to associate annotations with the objects in the video shared between 
participants of Ludwig's video conference for the benefit of expanding the annotation 
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capabilities of a video teleconference and increasing the clarity of communication 
between participants. Ludwig's "devices" (a thing made for a particular purpose) are 
anything appearing in his video images, ranging from charts and graphs to objects in 
participants video. Although this reference is not relied upon in the rejection of the 
claims, the examiner would like to call to applicant's attention that the annotation of 
objects in a moving video is known. 

On page 17, applicant argues that amended claim 24 is not taught by the cited 
references. The examiner disagrees; Claim 24, which now contains limitations similar 
to those of cancelled claim 33 as well as the additional limitation of annotations being 
transferred to a physical device in the live views if the annotation is at least partially 
drawn over the physical device as it appears in the live video image, is taught by Emens 
in view of Ayatsuka, Burt, and Schneiderman. As previously pointed out in the 
examiner's above discussion of claim 18, Schneiderman teaches annotating an "image 
aspect" (col. 7 II. 4-9), where an image aspect is defined as "a scene, object, event, 
design, etc" (col. 7 II. 23-25). A "device", by definition, encompasses any thing made for 
a particular purpose. Therefore any object that has a purpose qualifies as a device. 
The annotation applies to the image aspect which it is drawn over. 

Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
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invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

6. Claims 1-5, 9-14, 16-17, and 36-46 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ludwig et al (US 7,1 85,054 B1 ) in view of Okuno (US 6,977,672 B1 ) 
and further in view of Saka (US 2004/0070608 A1 ). 

Regarding claim 1 , Ludwig teaches a method for exchanging information 
in a shared interactive environment (abstract) comprising: 
a first live video image and a second live video image (figure 40 element 203 
shows multiple live video images) 

Ludwig does not teach selecting a first physical device in a first live video 
image wherein the first physical device has information associated with it; 
causing the information to be transferred to a second physical device in a second 
live video image wherein the transfer is brought about by manipulating a visual 
representation of the information in the first live video image and in the second 
live video image; 

wherein at least one of the first physical device and the second physical device 
has a statically or dynamically defined hotspot in the first live video image and or 
the second live video image; 

wherein the manipulation includes interacting with the first live video image and 
the second live video image; 

wherein the first physical device and the second physical device are part of the 
shared interactive environment; and 
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wherein the first physical device and the second physical device are not the 
same. 

Okuno teaches an interactive environment that includes selecting a first 
physical device in a first live video image wherein the first physical device has 
information associated with it (figure 17 and column 4 lines 23-32, figure 18 and 
column 4 lines 35-41, figure 19 and column 4 lines 45-52,); 
wherein manipulation includes interacting with the live video images (column 4 
lines 38-41 , figure 1 8). It would have been obvious to one of ordinary skill in the 
art at the time the invention was made to incorporate Okuno's teaching of using 
remote camera control and a live video image of a device in order to remotely 
manipulate/interact with said device with Ludwig's videoconferencing invention 
for the benefit of allowing video conference participants to control computers, 
printers, and other commonly used conference devices that are not physically 
reachable. 

Ludwig in view of Okuno does not teach causing the information to be 
transferred to a second physical device in a second live video image wherein the 
transfer is brought about by manipulating a visual representation of the 
information in the first live video image and in the second live video image; 
wherein at least one of the first physical device and the second physical device 
has a statically or dynamically defined hotspot in the first live video image and or 
the second live video image; 

wherein the first physical device and the second physical device are part of the 
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shared interactive environment; and 

wherein the first physical device and the second physical device are not the 
same. 

Saka teaches causing information to be transferred to a second physical 
device wherein the transfer is brought about by manipulating a visual 
representation of the information in a the first live video image (local desktop 
image) and in the second live video image (remote desktop image) (page 1 
section 001 1) (Ludwig, in col. 4 II. 60-65 teaches sharing screen snapshots such 
as the live desktop images used in Saka); 

wherein at least one of the first physical device and the second physical device 
has a statically or dynamically defined hotspot in the first live video image and or 
the second live video image (see icons in fig. 12); 

wherein the first physical device and the second physical device are part of the 
shared interactive environment (page 1 section 0011); and 
wherein the first physical device and the second physical device are not the 
same (page 1 section 001 1 ). It would have been obvious to one of ordinary skill 
in the art at the time the invention was made to use the information transfer 
interface taught by Saka in the videoconferencing with remote control of devices 
taught by Ludwig in view of Okuno for the benefit of providing videoconference 
participants with a convenient and simple (for participants) method of sharing 
information. For example, the action of sending data to the printer disclosed in 
Okuno (figure 10 shows an interface for controlling the printer, where the 
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structure of the commands to carry out the actions are shown in figure 9 list 
element 7 and figure 5) from a computer as disclosed in Saka would be carried 
out using Saka's "drag and drop" method. Alternatively, Okuno's printer could be 
replaced with one of Saka's remote machines so that control of the multiple 
computers disclosed in Saka's network could be initiated by the live video image 
manipulation taught by Okuno. 

Regarding claim 2, when read in light of claim 1, Saka further teaches the 
manipulation is accomplished by dragging the representation from the first 
physical device and dropping the representation on the second physical device 
(abstract). 

Regarding claim 3, when read in light of claim 1 , limitation "changes to the 
physical devices are visible to all participants in the shared interactive 
environment" is inherent. If all participants in the videoconference have the 
capabilities disclosed in Ludwig in view of Okuno and Saka, all participants would 
have the ability to focus on any physical device and obtain current associated 
information, which would reflect any changes. 

Regarding claim 4, when read in light of claim 1, Okuno further teaches a 
physical device can include a printer (figure 18). In regards to limitation "a 
physical device can include a display, a projector, a facsimile machine, a 
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personal digital assistant, a computer, and a portable computer", Okuno further 
discloses in column 5 lines 58-64 that commands can be executed on devices 
with addresses, such as IP addresses . It is well known in the art that displays, 
projectors, facsimile machines, personal digital assistants, computers, and 
portable computers can have IP addresses associated with them. 

Regarding claim 5, when read in light of claim 1 , Ludwig further teaches 
annotating at least one of the first live video image and the second live video 
image (fig 2b, col. 6 II. 57-61 . Updates/annotations are made to the image in real 
time). 

Regarding claim 9, when read in light of claim 1 , Okuno further teaches at 
least one of the first physical device and the second physical device has 
associated with it a pop-up control panel through which a user can configure and 
control it (figure 18, column 4 lines 38-41). 

Regarding claim 10, Ludwig teaches a method for exchanging information 
in a shared interactive environment (abstract), comprising: 
a first live video image and a second live video image (figure 40 element 203 
shows multiple live video images). 

Ludwig does not teach selecting a first object wherein the first object is 
one of: 1 ) a physical device in a first live video image and 2) an icon on a 



Application/Control Number: 10/629,403 Page 10 

Art Unit: 2423 

computing device; 

causing information associated with the first object to be transferred to a second 

object wherein the second object is the other of 1 ) the physical device in a first 

live video image; and 2) the icon on the computing device; 

wherein the first physical device has a dynamically defined hotpot in the first live 

video; 

wherein the transfer is brought about by manipulating a visual representation of 
the information; 

wherein the manipulation includes interacting with the first object and the second 
object; and 

wherein the physical device is part of the shared interactive environment. 

Okuno teaches selecting a first object wherein the first object is one of: 1 ) 
a physical device in a first live video image (figure 17 and column 4 lines 23-32, 
figure 18 and column 4 lines 35-41, figure 19 and column 4 lines 45-52); and 2) 
an icon on a computing device; 

wherein the manipulation includes interacting with the object in the live video 
(column 4 lines 38-41 , figure 1 8). It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to incorporate Okuno's teaching 
of using remote camera control and a live video image of a device in order to 
remotely manipulate/interact with said device with Ludwig's videoconferencing 
invention for the benefit of allowing video conference participants to control 



Application/Control Number: 10/629,403 Page 1 1 

Art Unit: 2423 

computers, printers, and other commonly used conference devices that are not 
physically reachable. 

Ludwig in view of Okuno does not teach causing information associated 
with the first object to be transferred to a second object wherein the second 
object is the other of: 1 ) a physical device in a first live video image; and 2) an 
icon on a computing device; 

wherein the first physical device has a dynamically defined hotpot in the 
first live video; 

wherein the transfer is brought about by manipulating a visual representation of 
the information; 

wherein the manipulation includes interacting with the first object and the second 
object; and 

wherein the physical device is part of the shared interactive environment. 

Saka teaches causing information associated with the first object to be 
transferred to a second object wherein the second object is the other of: 1 ) a 
physical device in a first live video image; and 2) an icon on a computing device 
(page 1 section 0011); 

wherein the first physical device has a dynamically defined hotpot in the first live 
video (see fig. 12 icons). 

wherein the transfer is brought about by manipulating a visual representation of 
the information (icon) in the first live video(page 1 section 0011); 
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wherein the manipulation includes interacting with the first object in the first live 
video image and the second object (page 1 section 001 1); and 
wherein the physical device is part of the shared interactive environment (page 1 
section 001 1 ). It would have been obvious to one of ordinary skill in the art at the 
time the invention was made to use the information transfer interface taught by 
Saka in the videoconferencing with remote control of devices taught by Ludwig in 
view of Okuno for the benefit of providing videoconference participants with a 
convenient and simple (for participants) method of sharing information. For 
example, the action of sending data to the printer disclosed in Okuno (figure 10 
shows an interface for controlling the printer, where the structure of the 
commands to carry out the actions are shown in figure 9 list element 7 and figure 
5) from a computer as disclosed in Saka would be carried out using Saka's "drag 
and drop" method. Alternatively, Okuno's printer could be replaced with one of 
Saka's remote machines so that control of the multiple computers disclosed in 
Saka's network could be initiated by the live video image manipulation taught by 
Okuno. 

Regarding claim 1 1 , when read in light of claim 1 0, Saka further teaches 
the manipulation is accomplished by dragging the representation from the first 
object and dropping the representation on the second object (abstract). 
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Regarding claim 12, when read in light of claim 10, limitation "changes to 
the physical device are visible to all participants in the shared interactive 
environment" is inherent. If all participants in the videoconference have the 
capabilities disclosed in Ludwig in view of Okuno and Saka, all participants would 
have the ability to focus on any physical device and obtain current associated 
information, which would reflect any changes. 

Regarding claim 1 3, when read in light of claim 1 0, Okuno further teaches 
a physical device can include a printer (figure 18). In regards to limitation "a 
physical device can include a display, a projector, a facsimile machine, a 
personal digital assistant, a computer, and a portable computer", Okuno further 
discloses in column 5 lines 58-64 that commands can be executed on devices 
with addresses, such as IP addresses . It is well known in the art that displays, 
projectors, facsimile machines, personal digital assistants, computers, and 
portable computers can have IP addresses associated with them. 

Regarding claim 14, when read in light of claim 1 , Ludwig further teaches 
annotating the first live video image (fig 2b, col. 6 II. 57-61 . Updates/annotations 
are made to the image in real time). 
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Regarding claim 16, when read in light of claim 10, Okuno further teaches 
the information can include a digital file (column 4 lines 25-29), a sound (column 
4 lines 29-32), and an audio/video presentation (column 4 lines 29-32). 

Regarding claim 17, when read in light of claim 10, Okuno further teaches 
the physical device has associated with it a pop-up control panel through which a 
user can configure and control it (figure 18, column 4 lines 38-41). 

Regarding claim 37, Ludwig, Okuno, and Saka all implement their 
inventions in hardware, which inherently requires machine readable instructions, 
therefore a computer readable memory embodying the method described in 
claim 10 is taught by Ludwig in view of Okuno and Saka (see analysis of claim 
10). 

Regarding claim 38, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer readable memory 
embodying the method described in claim 1 1 is taught by Ludwig in view of 
Okuno and Saka (see analysis of claim 1 1 ). 

Regarding claim 39, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
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machine readable instructions, therefore a computer data signal embodying the 
method described in claim 12 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 1 2). 

Regarding claim 40, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 13 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 1 3). 

Regarding claim 41 , when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 14 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 14). 

Regarding claim 42, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 1 5 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 1 5). 
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Regarding claim 43, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 10 with the additional limitation of claim 7 is taught by 
Ludwig in view of Okuno and Saka (see analysis of claims 10 and 7). 

Regarding claim 44, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 16 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 16). 

Regarding claim 45, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 17 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 1 7). 

Regarding claim 46 Ludwig, Okuno, and Saka all implement their 
inventions in hardware, and therefore teach a system with means for carrying out 
the method described in claim 1 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 1). 
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7. Claims 6, 8, and 15 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Ludwig et al (US 7,185,054 B1) in view of Okuno (US 6,977,672 B1) and Saka (US 
2004/0070608 A1 ), and further in view of Shneiderman (US 7,01 0,751 B2). 

Regarding claim 6, when read in light of claim 5, Ludwig in view of Okuno 
and Saka teaches the method of claim 5. 

Ludwig in view of Okuno and Saka does not teach automatically 
transferring the annotation to a physical device if the annotation is at least 
partially drawn over the physical device as it appears in a live video image. 

Shneiderman teaches automatically transferring the annotation to an 
"image aspect" (a physical device would qualify as an image aspect, col. 7 II. 23- 
25) in an image if the annotation is at least partially drawn over the physical 
device as it appears in a live video image (col. 7 II. 29-32, col. 4 II. 27-29, see fig. 
5 annotation of Rae Earnshaw). It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to annotate aspects of images 
for the benefit of making annotations more specific to certain attributes of an 
image rather than the entire image. 

Regarding claim 8, Ludwig in view of Okuno and Saka further teaches the 
information can include a digital file (Okuno column 4 lines 25-29), a sound 
(Okuno column 4 lines 29-32), and an audio/video presentation (Okuno column 4 
lines 29-32). 
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Ludwig in view of Okuno and Saka does not teach the information can 
include an annotation. 

Shneiderman teaches the information can include an annotation (page 3 
section 0055-0056). It would have been obvious to one of ordinary skill in the art 
at the time the invention was made to annotate aspects of images for the benefit 
of making annotations more specific to certain attributes of an image rather than 
the entire image. 

Regarding claim 15, Ludwig in view of Okuno and Saka teaches the 
method of claim 14. 

Ludwig in view of Okuno and Saka does not teach automatically 
transferring the annotation to the physical device if the annotation is at least 
partially drawn over the physical device as it appears in a live video image. 

Shneiderman teaches automatically transferring the annotation to an 
"image aspect" (a physical device would qualify as an image aspect, col. 7 II. 23- 
25) in an image if the annotation is at least partially drawn over the physical 
device as it appears in a live video image (col. 7 II. 29-32, col. 4 II. 27-29, see fig. 
5 annotation of Rae Earnshaw). It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to annotate aspects of images 
for the benefit of making annotations more specific to certain attributes of an 
image rather than the entire image. 
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8. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Ludwig et 
al (US 7,185,054 B1) in view of Okuno (US 6,977,672 B1) and Saka (US 2004/0070608 
A1 ), and further in view of Emens et al (US 6,463,343 B1 ). 

Regarding claim 7, when read in light of claim 1 , Ludwig in view of Okuno 
and Saka teach the method of claim 1 . 

Ludwig in view of Okuno and Saka does not teach that the first live video 
image and the second live video image are the same. 

Limitation "the first live video image and the second live video image are 
the same", in light of claim 1 limitation "selecting a first physical device in a first 
live video image 

causing the information to be transferred to a second physical device in a second 
live video image wherein the transfer is brought about by manipulating a visual 
representation of the information" is taught by Emens column 1 lines 57-60 and 
figure 2c, in which a user can select a device to control amongst a plurality of 
devices in a single live image. It would have been obvious to one of ordinary skill 
in the art at the time the invention was made to incorporate Emen's teaching of 
being able to select a device to remotely control from a plurality of devices in a 
single live image with the invention taught by Ludwig in view of Okuno and Saka. 
This combination of inventive ideas would enable videoconference participants to 
remotely control devices in a room without adjusting the camera angle to zoom in 
on each device. 
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9. Claims 18 and 21 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Ludwig et al (US 7,185,054 B1) in view of Shneiderman (US 7,010,751 B2). 

Regarding claim 18, Ludwig teaches a method for annotating a live video 
image wherein the annotation is visible to at least one participant in a shared 
interactive 

environment (fig 2b, col. 6 II. 57-61 . Updates/annotations are made to the image 
in real time). 

Ludwig does not teach annotating the live video image and automatically 
transferring the annotation to a physical device in the live video image if the 
annotation is at least partially drawn over the physical device as it appears in a 
live video image. 

Shneiderman teaches a method for annotating video image comprising 
annotating the live video image and automatically transferring the annotation to a 
physical device if the annotation is at least partially drawn over the physical 
device as it appears in a live video image (col. 7 II. 29-32, col. 4 II. 27-29, see fig. 
5 annotation of Rae Earnshaw). It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to provide the annotation 
capabilities of Shneiderman in the videoconferencing system and method taught 
by Ludwig for the benefit of easily and clearly communicating information about 
image aspects in a videoconferencing image to other participants. 
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Regarding claim 21 , when read in light of claim 1 8, Ludwig in view of 
Shneiderman teaches the method of claim 18. 

Ludwig in view of Shneiderman does not teach the live video image is one 
of: a panoramic view and a zoomed view. 

Official notice is taken that panning and zooming are well known and 
widely used tools in video surveillance and conferencing systems. It would have 
been obvious to one of ordinary skill in the art at the time the invention was made 
to include panning and zooming capabilities in the videoconferencing system 
taught by Ludwig in view of Shneiderman for the benefit of allowing 
videoconference participants to have access to view of any resources in the 
collective rooms, such as whiteboards or illustrative models. 

10. Claims 19-20 and 22-23 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ludwig et al (US 7,1 85,054 B1 ) in view of Shneiderman (US 
7,01 0,751 B2), and further in view of Okuno (US 6,977,672 B1 ). 

Regarding claim 19, when read in light of claim 18, Ludwig in view of 
Shneiderman teaches the method of claim 18. 

Ludwig in view of Shneiderman does not teach at least one participant can 
interact with the physical device. 

Okuno teaches an video in which a viewed physical devices can be controlled 
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remotely by a participant (column 4 lines 38-41 , figure 18). It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to 
incorporate Okuno's teaching of using remote camera control and a live video 
image of a device in order to remotely manipulate/interact with devices that are 
not physically reachable. 

Regarding claim 20, when read in light of claim 18, Ludwig further teaches 
each of the at least one participants can interact with the shared interactive 
environment through different computing devices (col. 6 II. 28-51). 

Regarding claim 22, when read in light of claim 18, Ludwig in view of 
Shneiderman and Okuno further teaches the physical device can include a 
display, a projector, a printer, a facsimile machine, a personal digital assistant, a 
computer, and a portable computer. Okuno discloses in column 5 lines 58-64 
that commands can be executed on devices with addresses, such as IP 
addresses. It is well known in the art that displays, projectors, facsimile 
machines, personal digital assistants, computers, and portable computers can 
have IP addresses associated with them. 



Regarding claim 23, when read in light of claim 18, Ludwig in view of 
Shneiderman and Okuno further teaches the physical device has associated with 
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it a pop-up control panel through which a user can configure and control it 
(Okuno figure 18, column 4 lines 38-41). 

1 1 . Claims 24, 26-27, 30, 32, and 34-35 are rejected under 35 U.S.C. 1 03(a) as 
being unpatentable over Emens et al (US 6,463,343 B1) in view of Ayatsuka et al (US 
7,188,139 B1), and further in view of Burt et al. "Object tracking with a moving camera", 
IEEE An Application of Dynamic Motion Analysis. 1989 pp. 2-12, and Schneiderman 
(US 7,010,751 B2). 

Regarding claim 24, Emens teaches a shared interactive environment, 
comprising: 

a camera system to provide a first live view and second live view (col. 3 II. 8-9, 
where the first live view and second live view are the same in this instance; col. 5 
II. 15-20 teaches multiple cameras for providing multiple different views, or a 
moving camera for providing different views from one camera); 
a first graphical user interface (GUI) coupled to the camera system and to 
present the first live view and the second live view, wherein the views can 
capture a physical device (abstract); 

a device controller to dynamically control the physical device in response to 
interaction of a first user with the GUI. 

Emens does not teach the second live view can be configured to zoom in 
on a portion of the first live view; 
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wherein the interaction can including annotating at least one of 1) the first live 
view; and 2) the second live view (fig. 2e, col. 3 II. 10-11); 
wherein annotations are automatically transferred to the physical device in the 
live views if the annotation is at least partially drawn over the physical device as it 
appears in the live video; 

a device tracker coupled to the camera system and to dynamically recognize new 
physical devices; and 

wherein the camera system can be mounted on a mobile, robotic platform. 

In regards to limitation "wherein the second live view can be configured to 
zoom in on a portion of the first live view", official notice is taken that zooming is 
a well known and widely used tool in video surveillance and conferencing 
systems. It would have been obvious to one of ordinary skill in the art at the time 
the invention was made to include panning and zooming capabilities in the 
interactive environment taught by Emens for the benefit of allowing 
videoconference participants to have access to view of any resources in the 
collective rooms, such as whiteboards or illustrative models. 

Ayatsuka teaches a camera connected to a computer that can detect and 
recognize devices in the camera's view, and allows the viewed device to be 
controlled by the computer (abstract). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to incorporate the 
device detection capabilities taught by Ayatsuka in the device control system 
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taught by Emens for the benefit of allowing a user to register devices for control 
by a computer system without manually entering information for each device. 

Burt teaches a robotic camera that detects and tracks a target (abstract). 
It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to incorporate Burt's teaching of a robotic camera that keeps 
targeted objects in its line of view in the interactive environment taught by Emens 
in view of Ayatsuka for the benefit of maintaining a view of devices that the user 
may want to exercise control over. Burt's abstract teaches the application of this 
concept to automated surveillance in the abstract. 

Regarding limitations "wherein the interaction can including annotating at least 
one of 1 ) the first live view; and 2) the second live view (fig. 2e, col. 3 II. 10-11 ); 
wherein annotations are automatically transferred to the physical device in the 
live views if the annotation is at least partially drawn over the physical device as it 
appears in the live video", Schneiderman teaches annotating an "image aspect" 
(col. 7 II. 4-9), where an image aspect is defined as "a scene, object, event, 
design, etc" (col. 7 II. 23-25). A "device", by definition, encompasses any thing 
made for a particular purpose. Therefore any object that has a purpose qualifies 
as a device. The annotation applies to the image aspect which it is drawn over. 
It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include annotating capabilities in the live views shown in 
Emens for the benefit of labeling and identifying the various devices shown in 
Emen's user interface. 
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Regarding claim 26, when read in light of claim 24, Emens further teaches 
the device controller can control the physical device through at least one of: 1 ) an 
infrared communication channel; and 2) one or more networks (fig. 1 col. 5 II. 26- 
31). 

Regarding claim 27, when read in light of claim 24, Emens in view of 
Ayatsuka and Burt further teaches the device tracker can recognize new physical 
devices by at least one of: 1 ) image pattern recognition (Burt pg. 3 col. 2 
paragraph 2, fig. 2); 2) radio frequency transmission; and 3) acoustic signal. 

Regarding claim 30, when read in light of claim 24, Emens further teaches 
the GUI is implemented as one or more web pages (col. 2 II. 60-65). 

Regarding claim 32, when read in light of claim 24, Emens further teaches 
the physical device has a pop-up control panel that can be made apparent to the 
first user through the first GUI and wherein the pop-up control panel allows the 
first user to control and configure the physical device (fig. 2e). 

Regarding claim 34, when read in light of claim 24, both Emens and 
Ayatsuka teach software based control schemes for controlling objects and 
devices. It is inherent that the physical device can be represented by a set of 
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attributes and a set of behaviors, as those are the only ways of representing an 
object to a computer program. 

Regarding claim 35, when read in light of claim 34, Emens in view of 
Ayatsuka and Burt teach the shared interactive environment of claim 34. 

Emens in view of Ayatsuka and Burt does not teach the representation of 
the physical device is part of a device hierarchy. 

Both Emens and Ayatsuka teach software based control schemes for 
controlling objects and devices. Object oriented programming is an obvious 
approach to developing such software, as each device has its own attributes and 
control routines. The concept of classes and inheritance are well known and 
often used to create programs that support a variety of objects. Official notice is 
taken that it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to use objects and inheritance to implement device 
control software for the benefit of optimizing the software and reducing 
redundancy in the application programming for devices that may share similar 
functions. 

12. Claim 25 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1 ) in view of Ayatsuka et al (US 7,1 88,1 39 B1 ) and Burt et al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
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Analysis. 1989 pp. 2-12, and Schneiderman (US 7,010,751 B2), and further in view of 
Andersson (US 2002/01 1 1 999 A1 ). 

Regarding claim 25, when read in light of claim 24, Emens in view of 
Ayatsuka and Burt teaches the interactive environment of claim 24 and suggests 
implementing device control in a video teleconferencing environment (fig. 5), but 
does not disclose multiple users having access to the system (i.e. the first GUI 
allows the first user to interact the physical device; and wherein the interaction of 
the first user is apparent to a second user via a second GUI). 

Andersson teaches a system that allows networked computers to access 
and control devices connected to any computer on that network, where both host 
and remote computers can monitor device events (pg. 2 sect. 0019, 0023). It 
would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the remote access and device monitoring 
capabilities taught by Andersson in the shared interactive environment taught by 
Emens in view of Ayatsuka for the benefit of enhancing collaboration and making 
a more realistic environment for users who may share resources. 

13. Claim 28 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1) in view of Ayatsuka et al (US 7,188,139 B1)and Burtet al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
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Analysis. 1989 pp. 2-12, and Schneiderman (US 7,010,751 B2), and further in view of 
Hildebrandt (US 2004/0070616). 

Regarding claim 28, when read in light of claim 24, Emens in view of 
Ayatsuka and Burt teaches the interactive environment of claim 24, wherein the 
physical device can be a display (Ayatsuka fig. 5, col. 22 II. 63-67). 
Emens in view of Ayatsuka and Burt does not teach the display can include an 
image stack. 

Hildebrandt teaches an archive memory for use with an electronic 
whiteboard in which images associated with the whiteboard are grouped and 
saved. These images are stored as a stack, where the oldest image is deleted 
when a new image is added (see fig. 22 el 524). It would have been obvious to 
one of ordinary skill in the art at the time the invention was made to store a 
plurality of images captured from a display for the benefit of allowing a user to 
continually use a display resource while stil being able to access data previously 
represented on that display. 

14. Claim 29 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1 ) in view of Ayatsuka et al (US 7,1 88,1 39 B1 ) and Burt et al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
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Analysis. 1989 pp. 2-12, and Schneiderman (US 7,010,751 B2), and Andersson (US 
2002/01 1 1 999 A1 ), and further in view of Westfield (US 6,677,979 B1 ). 

Regarding claim 29, when read in light of claim 25, Emens in view of 
Ayatsuka, Burt, and Andersson teaches the shared interactive environment of 
claim 25. 

Emens in view of Ayatsuka, Burt, and Andersson does not teach the first 
GUI can provide a second live view that is different from the second live view 
provided by the second GUI. 

Ayatsuka suggests one environment in which devices remotely controlled 
through a camera view could be used is in a teleconference (fig. 5). 

Westfield teaches a teleconference environment in which a participant can 
see 2 views, where the second view can be specified for each participant 
(abstract, fig 8b, fig 7 and col. 2 II. 46-56). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to include the multiple 
views of an environment taught by Westfield in the interactive environment taught 
by Emens in view of Ayatsuka, Burt, and Andersson for the benefit of more 
closely mimicking human vision by providing both wide angle and focused views 
(col. 1 II. 43-54). 

15. Claim 31 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1 ) in view of Ayatsuka et al (US 7,1 88,1 39 B1 ) and Burt et al. 
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"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
Analysis. 1989 pp. 2-12, and Schneiderman (US 7,010,751 B2), and further in view of 
Westfield (US 6,677,979 B1). 

Regarding claim 31 , when read in light of claim 24, Emens in view of 
Ayatsuka and Burt teaches the shared interactive environment of claim 24. 

Emens in view of Ayatsuka and Burt does not teach the first user can 
select the second live view by drawing a diagonal in the first live view. 
Westfield teaches a teleconference environment in which a participant can see 2 
views, where the second view can be specified by a participant dragging a 
rectangle across the image in the first view, (abstract, fig 8b, fig 7 and col. 6 II. 
38-40). It would have been obvious to one of ordinary skill in the art at the time 
the invention was made to include the zoomed view of an environment taught by 
Westfield in the interactive environment taught by Emens in view of Ayatsuka, 
Burt, and Andersson for the benefit of more closely mimicking human vision by 
providing both wide angle and focused views (col. 1 II. 43-54). 

Conclusion 

2. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

3. Lennon (US 6,71 1 ,590 B1 ) teaches a method of linking metadata, such as an 
annotation, to objects in a moving video. 
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4. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JASMINE STOKELY-COLLINS whose telephone 
number is (571) 270-3459. The examiner can normally be reached on M-Th 9:30-5:00 
EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Andrew Koenig can be reached on (571) 272-7296. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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Examiner, Art Unit 2423 
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Supervisory Patent Examiner, Art Unit 2423 



